Stretching comms to the limit in the wake of a natural disaster

The effects of a brutal storm that hit the eastern Spanish peninsula in late October were felt beyond the affected areas. Videos shared on social media and calls to television and radio programs showed the magnitude of the rain and flooding in Valencia, Albacete, and other towns in the area. Many of these testimonies had several things in common: I can’t find another person, they don’t pick up the phone at emergency services, I don’t know how much battery I have left or if I’ll be able to call again.

In the midst of all the tragedy, the need to ask for or manage help, locate people, or simply tell what was happening depended on the communication systems continuing to function reliably as well as they do in normal circumstances.

But the severity of the floods, combined with strong winds and isolated tornadoes, further complicated the situation. “This led to significant damage to telecom infrastructure,” says Rubén Nicolás-Sans, vice-dean of science and technology at UNIE University, “particularly where the water covered cables and stations, causing interruptions in both mobile and landline telephony.” Added to this were power outages that turned off routers and left devices unable to recharge their batteries. “It interrupted essential services, including data centers that support digital communications, limiting the ability to communicate between citizens and emergency services, especially in the first hours of the crisis,” he adds.  

The experience of local operators

Trying to restore service as soon as possible wasn’t easy. The roads were difficult to navigate, when they could be used at all, and in those first hours, no one knew what the technical teams would find. This was the situation faced by internet service provider Fibra Valencia, which works in the most affected areas. CEO Antonio Costa says the alarms in its data centres went off the night the floods started, and they tried to move a team there, but when they arrived, they found the water was already too high and they had to leave. The next day, they began working and found the damage extended to the fiber. But after about 10 hours, one team raised the entire data center, and another team managed to pass the fiber over a damaged bridge.

Valencia wasn’t the only region affected. Letur, in Albacete where Excom operates, was especially hit. Ginés Martínez, a technician with the company, was transferred from Murcia to support service recovery. He explains that Excom warned beforehand of possible incidents and, within 48 hours, service was restored to the town hall and health centre. Coverage was also provided to temporary huts on the street for the Civil Protection, Red Cross, and other teams, and WiFi connection points were set up for citizens. A full recovery will be another thing, Martínez adds, which depends on repair to roads and other infrastructure.   

The work of the big telecos 

The situation has been closely monitored by the country’s main operators. As Julia Velasco, director of network technology and operations at Vodafone, recalls, there are also people from her and other teams who work and live in affected areas. With these storms, Velasco and her team followed the situation for some time and eventually established a war room. “We went into crisis mode,” she says. The operations teams met to work in a coordinated manner and contacted the company that helps with incident management on the ground, and mobilized to try to reinforce teams, even with people from neighboring areas.  

At the time, no one was sent to the field for safety reasons, but the next day, the reality of the damage was clear. “The main problem we faced as a mobile team was physically getting to the sites where equipment had to be changed,” says Velasco. “Transport infrastructures were most damaged because the fiber is usually deployed close to civil infrastructures like roads, bridges, and high voltage towers.” The force of the floods broke part of the pipelines where the fiber was, and detecting all the cuts was a very costly and laborious task. 

At MasOrange, alarm bells started ringing when conditions deteriorated, and they decided to activate their incident management protocol. “First, more distant or dispersed sites fell and then we began to notice that large power plants fell as well,” says Manuel Muñoz, their director of operations. “That’s when you could already see the impact, which on the night of the first day was already a lot.” Resources and material were then prepared in nearby areas.  

At first, the emergency services didn’t let them through, or made them leave, so they focused on assessing the impact and, the next day, triaging the most affected infrastructures. “There were isolated villages with all its antennas down,” he says. Here the priority was to put at least one back in place, so some service could be guaranteed. “In some places it was quite complex, so we also moved satellite units to allow us, or at least the security forces, to have communication,” he adds.  

MasOrange was able to eventually recover more than 98% of service, and they continue to install new fiber optic cables and erect portable mobile stations. Vodafone estimates that 97% of mobile lines have been recovered and 91% of fixed lines. And so far, they’ve sent more than 250 network technicians to affected areas.

At Telefónica, it was its own CEO, José María Álvarez-Pallete, who a week after the first impact of the storm published an open letter in which he outlined the extent of the damage: more than 250 base stations, several network nodes, several fiber exchanges, and part of the network itself. In total, 30% of the fixed network and 22% of the mobile network in Valencia were affected. The company also reported that the mobile network had been fully recovered, and work was underway to check fixed infrastructure to ensure it was also working at 100%. At first, work was done to guarantee service to emergency teams, followed by service to citizens.  

Collaboration for the greater good 

“Our sector is one in which we compete on a commercial level, and collaborate on the technical side,” Velasco says. “And in this situation, even more so.” Velasco and Muñoz both speak of coordinating to organize forces, with daily contact, and work with local teams to be able to access certain points or fix problems.

This collaboration hasn’t been limited to the big operators, though. Costa claims the role of regional ones, such as Airsip, PTV or Fibra Valencia, were first to reach places and restore services. “In the end, the priority was for the mobile telephony part to work, because it was distressing to be totally cut off from communication,” he says.  

Companies have launched other measures to help communications among affected citizens, such as unlimited data, greater capacity, or gigabyte extensions. MasOrange, for example, also offers free telemedicine services, psychological support, and legal and insurance advice, and Vodafone offers up to three days a year of paid leave to participate in relief work.  

Many organizations have also launched a solidarity initiative to help recover critical information from damaged company equipment. The government, among other measures for the general population, reached an agreement with telecommunications companies so people affected by power outages will be compensated automatically in their next bills, without having to request it. The Generalitat, the government of Catalonia, has also opened the WiFi networks of primary schools in Valencia to facilitate connection. 

Lessons learned

A full recovery is expected to take a long time, as are measures implemented to mitigate damage after future events. Nicolás-Sans says there must be more exhaustive prior planning and rapid responses required. “It’s crucial that solid contingency plans are in place that include both mobile and satellite emergency communication systems,” he says, in addition to repair and restoration protocols for critical infrastructures like telecom networks and data centers. He also points out infrastructure redundancy. “Operators must invest in more resilient networks, such as underground systems or flood protection solutions,” he says. “And the use of backup technologies, including power generators and protection systems for telecommunications equipment, is essential.” So this moment is being used as an opportunity to make improvements to facilities and be more prepared.

These tasks will take time, but companies are working to consolidate solutions, especially as the severity and frequency of damaging weather is becoming more common. The work against the clock continues.